Skip to content

fix(server): reconcile crash-frozen running turns to interrupted on boot (#5) - #13

Merged
radroid merged 1 commit into
mainfrom
t3x/crash-recovery-reconcile-running-turns
Jul 25, 2026
Merged

fix(server): reconcile crash-frozen running turns to interrupted on boot (#5)#13
radroid merged 1 commit into
mainfrom
t3x/crash-recovery-reconcile-running-turns

Conversation

@radroid

@radroid radroid commented Jul 25, 2026

Copy link
Copy Markdown
Owner

Summary

Fixes #5. When the desktop backend exits ungracefully (SIGKILL / crash / OOM — no SIGTERM), in-flight agent turns are left frozen at projection_turns.state = "running" with projection_thread_sessions.active_turn_id still set. Nothing reconciles them on next launch, so on a later resume the provider reports the cut-off turn as cancelled/aborted, and auto-resume never recovers them. A clean quit does the right thing via an Effect finalizer (ProviderService.runStopAlladapter.stopAll() → a session-set that settles running turns to interrupted); only ungraceful exit was broken.

Approach

At fresh boot no turn is actually live — adapter session maps start empty and runtime rows carry no pid/epoch — so every leftover in-flight turn in the read model is by definition a crash remnant. The fix adds a one-shot boot reconciler that mirrors the graceful path exactly:

  • New apps/server/src/orchestration/Layers/CrashRecoveryReconciler.tsreconcileInterruptedTurnsOnBoot reads the read model (ProjectionSnapshotQuery.getSnapshot()), finds threads with a leftover in-flight turn (latestTurn.state === "running", or session status ∈ {running, starting}, or non-null activeTurnId), and for each dispatches a thread.session.set command with status: "stopped", activeTurnId: null (mirroring what ProviderRuntimeIngestion dispatches on session.exited). This settles the running turn to the canonical resumable interrupted state and clears active_turn_id.
  • Wired into serverRuntimeStartup.ts as a startup phase before reactors.start and before command-ready is signalled, so no reactor or user command observes the stale state, and the (now-unblocked) ProviderSessionReaper can idle-sweep the runtime row.

Key correctness property: reconciliation goes through the event log (a dispatched command), never a direct projection write — projections are a pure function of the event log and a direct write would be reverted on rebuild. It is fully error-swallowed (never error channel) so a bad thread or persistence hiccup can't block boot.

Scope (deliberate)

Reconciling to a resumable interrupted state satisfies the issue's primary acceptance ("or at minimum settled to a resumable interrupted state"). Auto-resume re-arming (auto-continuing the turn) needs a new crash marker + a boot-time producer to distinguish crash-interrupted from user/stop-interrupted turns; that's a documented, separate follow-up (noted in code comments) rather than bundled here.

Tests (TDD — red first)

  • CrashRecoveryReconciler.test.ts (new, integration over the real engine + projection pipeline + in-memory sqlite): precondition (turn running, session running, active_turn_id set) → reconcile → turn interrupted with completed_at, session stopped, active_turn_id null, reconciledCount === 1; idempotency (second run is a no-op); multi-thread selectivity (only crashed threads change; already-settled untouched); event-log backing (a new thread.session-set event is appended, surviving a rebuild).
  • serverRuntimeStartup.test.ts (extended): the reconcile phase runs before reactors.start / command-ready, and a dispatch failure is swallowed (startup still completes).
  • t3x/autoResume/guards.test.ts (extended): a settled interrupted + stopped thread is not threadIsProgressing (documents resumability).

Verification

  • New/extended trio → 33 pass; full src/orchestration/ + src/t3x/autoResume/255 pass.
  • tsgo --noEmit → exit 0, no errors.
  • vp fmt --check on changed files → clean.

Note: src/t3x/autoResume/Reactor.test.ts has a pre-existing timing-flaky case (unchanged by this PR — verified identical to main); unrelated to this change.

🤖 Generated with Claude Code

… boot

Fixes #5.

When the backend exits ungracefully (SIGKILL/crash/OOM, no SIGTERM), the
Effect finalizer a clean quit runs (ProviderService.runStopAll ->
adapter.stopAll() -> a session-set that settles running turns to
"interrupted") never fires. In-flight turns are left frozen at
projection_turns.state="running" with active_turn_id still set; on a later
resume the provider reports the cut-off turn as cancelled/aborted, and
auto-resume never recovers them.

At fresh boot no turn is actually live (adapter session maps start empty;
runtime rows carry no pid/epoch), so every leftover in-flight turn is a crash
remnant. Add a one-shot boot reconciler that mirrors the graceful path: for
each such thread it dispatches a stopped `thread.session.set` command (through
the event log, not a direct projection write), settling the running turn to
the resumable "interrupted" state and clearing active_turn_id. Wired as a
startup phase before reactors start and before commands are accepted.

Scope: reconciling to a resumable interrupted state satisfies the issue's
primary acceptance. Auto-resume RE-ARMING (a crash marker + boot producer to
auto-continue the turn) is left as a documented follow-up.

Adds CrashRecoveryReconciler.test.ts (precondition, reconcile, idempotency,
multi-thread selectivity, event-log backing) and extends serverRuntimeStartup
and autoResume/guards tests.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jul 25, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: c4e9af4f-873f-4aac-99a2-0abf74fde18b

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch t3x/crash-recovery-reconcile-running-turns

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@radroid
radroid merged commit 488776d into main Jul 25, 2026
1 check passed
@radroid
radroid deleted the t3x/crash-recovery-reconcile-running-turns branch July 30, 2026 14:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ungraceful backend exit leaves in-flight turns frozen as running → surface as cancelled; auto-resume doesn't recover them

1 participant